Identifying the items most relevant to a current query based on items selected in connection with similar queries

ABSTRACT

The present invention provides a software facility for identifying the items most relevant to a current query based on items selected in connection with similar queries. In preferred embodiments of the invention, the facility receives a query specifying one or more query terms. In response, the facility generates a query result identifying a plurality of items that satisfy the query. The facility then produces a ranking value for at least a portion of the items identified in the query result by combining the relative frequencies with which users selected that item from the query results generated from queries specifying each of the terms specified by the query. The facility identifies as most relevant those items having the highest ranking values.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part of U.S. patent applicationSer. No. 09/665,822 filed Sep. 20, 2000, which is a continuation-in-partof U.S. patent application Ser. No. 09/041,081 filed Mar. 10, 1998 nowissued as U.S. Pat. No. 6,185,558, which is a continuation-in-part ofU.S. patent application Ser. No. 09/033,824 filed Mar. 3, 1998, nowabandoned, all of which are hereby incorporated by reference in theirentirety.

TECHNICAL FIELD

The present invention is directed to the field of query processing.

BACKGROUND OF THE INVENTION

Many World Wide Web sites permit users to perform searches to identify asmall number of interesting items among a much larger domain of items.As an example, several web index sites permit users to search forparticular web sites among most of the known web sites. Similarly, manyonline merchants, such as booksellers, permit users to search forparticular products among all of the products that can be purchased froma merchant. In many cases, users perform searches in order to ultimatelyfind a single item within an entire domain of items.

In order to perform a search, a user submits a query containing one ormore query terms. The query also explicitly or implicitly identifies adomain of items to search. For example, a user may submit a query to anonline bookseller containing terms that the user believes are words inthe title of a book. A query server program processes the query toidentify within the domain items matching the terms of the query. Theitems identified by the query server program are collectively known as aquery result. In the example, the query result is a list of books whosetitles contain some or all of the query terms. The query result istypically displayed to the user as a list of items. This list may beordered in various ways. For example, the list may be orderedalphabetically or numerically based on a property of each item, such asthe title, author, or release date of each book. As another example, thelist may be ordered based on the extent to which each identified itemmatches the terms of the query.

When the domain for a query contains a large number of items, it iscommon for query results to contain tens or hundreds of items. Where theuser is performing the search in order to find a single item,application of conventional approaches to ordering the query resultoften fail to place the sought item or items near the top of the queryresult, so that the user must read through many other items in the queryresult before reaching the sought item. In view of this disadvantage ofconventional approaches to ordering query results, a new, more effectivetechnique for automatically ordering query results in accordance withcollective and individual user behavior would have significant utility.

Further, it is fairly common for users to specify queries that are notsatisfied by any items. This may happen, for example, where a usersubmits a detailed query that is very narrow, or where a user mistypesor misremembers a term in the query. In such cases, conventionaltechniques, which present only items that satisfy the query, present noitems to the user. When no items are presented to a user in response toissuing a query, the user can become frustrated with the search engine,and may even discontinue its use. Accordingly, a technique fordisplaying items relating to at least some of the terms in a query evenwhen no items completely match the query would have significant utility.

In order to satisfy this need, some search engines adopt a strategy ofeffectively automatically revising the query until a non-empty resultset is produced. For example, a search engine may progressively deleteconjunctive, i.e., ANDed, terms from a multiple term query until theresult set produced for that query contains items. This strategy has thedisadvantage that important information for choosing the correct itemscan be lost when query terms are arbitrarily deleted. As a result, thefirst non-empty result set can be quite large, and may contain a largepercentage of items that are irrelevant to the original query as awhole. For this reason, a more effective technique for displaying itemsrelating to at least some of the terms in a query even when no itemscompletely match the query would have significant utility.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level block diagram showing the computer system uponwhich the facility preferably executes.

FIG. 2 is a flow diagram showing the steps preferably performed by thefacility in order to generate a new rating table.

FIGS. 3 and 4 are table diagrams showing augmentation of an item ratingtable in accordance with step 206 (FIG. 2).

FIG. 5 is a table diagram showing the generation of rating tables forcomposite periods of time from rating tables for constituent periods oftime.

FIG. 6 is a table diagram showing a rating table for a composite period.

FIG. 7 is a flow diagram showing the steps preferably performed by thefacility in order to identify user selections within a web server log.

FIG. 8 is a flow diagram showing the steps preferably performed by thefacility to order a query result using a rating table by generating aranking value for each item in the query result.

FIG. 9 is a flow diagram showing the steps preferably performed by thefacility to select a few items in a query result having the highestranking values using a rating table.

FIGS. 10-13 are display diagrams showing examples of considerations usedby embodiments of the facility to determine the level of effort expendedby the user to select an item from a query result.

DETAILED DESCRIPTION

A software facility (“the facility”) for identifying the items mostrelevant to a current query based on items selected in connection withsimilar queries is described. The facility preferably, generates rankingvalues for items indicating their level of relevance to the currentquery, which specifies one or more query terms. The facility generates aranking value for an item by combining rating scores, produced by arating function, that each correspond to the level of relevance of theitem to queries containing one of the ranking values. The ratingfunction preferably retrieves a rating score for the combination of anitem and a term from a rating table generated by the facility. Thescores in the rating table preferably reflect, for a particular item andterm, how often users have selected the item when the item has beenidentified in query results produced for queries containing the term. Insome embodiments, the scores also reflect the level of effort users werewilling to expend in order to find and select the selected items withinquery results.

In different embodiments, the facility uses the rating scores to eithergenerate a ranking value for each item in a query result, or generateranking values for a smaller number of items in order to select a fewitems having the top ranking values. To generate a ranking value for aparticular item in a query result, the facility combines the ratingscores corresponding to that item and the terms of the query. Inembodiments in which the goal is to generate ranking values for eachitem in the query result, the facility preferably loops through theitems in the query results and, for each item, combines all of therating scores corresponding to that item and any of the terms in thequery. On the other hand, in embodiments in which the goal is to selecta few items in the query result having the largest ranking values, thefacility preferably loops through the terms in the query, and, for eachitem, identifies the top few rating scores for that term and any item.In some embodiments, the facility uses stemming techniques toincorporate scores for terms having the same roots as the terms in thequery. The facility then combines the scores identified for each item togenerate ranking values for a relatively small number of items, whichmay include items not identified in the query result. Indeed, theseembodiments of the invention are able to generate ranking values for anddisplay items even in cases in which the query result is empty, i.e.,when no items completely satisfy the query.

Once the facility has generated ranking values for at least some items,the facility preferably orders the items of the query result indecreasing order of ranking value. The facility may also use the rankingvalues to subset the items in the query result to a smaller number ofitems. By ordering and/or subsetting the items in the query result inthis way in accordance smith collective and individual user behaviorrather than in accordance with attributes of the items, the facilitysubstantially increases the likelihood that the user will quickly findwithin the query result the particular item or items that he or sheseeks. For example, while a query result for a query containing thequery terms “human” and “dynamic” may contain a book about humandynamics and a book about the effects on human beings of particledynamics, selections by users from early query results produced forqueries containing the term “human” show that these users select thehuman dynamics book much more frequently than they select the particledynamics book. The facility therefore ranks the human dynamics bookhigher than the particle dynamics book, allowing users, most of whom aremore interested in the human dynamics book, to select it more easily.This benefit of the facility is especially useful in conjunction withthe large, heterogeneous query results that are typically generated forsingle-term queries, which are commonly submitted by users.

Various embodiments of the invention base rating scores on differentkinds of selection actions performed by the users on items identified inquery results. These include whether the user displayed additionalinformation about an item, how much time the user spent viewing theadditional information about the item, how many hyperlinks the userfollowed within the additional information about the item, whether theuser added the item to his or her shopping basket, and whether the userultimately purchased the item. Embodiments of the invention alsoconsider selection actions not relating to query results, such as typingan item's item identifier rather than choosing the item from a queryresult. Additional embodiments of the invention incorporate into theranking process information is about the user submitting the query bymaintaining and applying separate rating scores for users in differentdemographic groups, such as those of the same sex, age, income, orgeographic category. Certain embodiments also incorporate behavioralinformation about specific users. Further, rating scores may be producedby a rating function that combines different types of informationreflecting collective and individual user preferences. Some embodimentsof the invention utilize specialized strategies for incorporating intothe rating scores information about queries submitted in different timeframes.

FIG. 1 is a high-level block diagram showing the computer system uponwhich the facility preferably executes. As shown in FIG. 1, the computersystem 100 comprises a central processing unit (CPU) 110, input/outputdevices 120, and a computer memory (memory) 130. Among the input/outputdevices is a storage device 121 such as a hard disk drive; acomputer-readable media drive 122, which can be used to install softwareproducts, including the facility, which are provided on acomputer-readable medium, such as a CD-ROM; and a network connection 123for connection the computer system 100 to other computer systems (notshown). The memory 130 preferably contains a query server 131 forgenerating query results from queries, a query result ranking facility132 for automatically ranking the items in a query result in accordancewith collective user preferences, and item rating tables 133 used by thefacility. While the facility is preferably implemented on a computersystem configured as described above, those skilled in the art willrecognize that it may also be implemented on computer systems havingdifferent configurations.

The facility preferably generates a new rating table periodically, and,when a query result is received, uses the last-generated rating table torank the items in the query result. FIG. 2 is a flow diagram showing thesteps preferably performed by the facility in order to generate a newrating table. In step 201, the facility initializes a rating table forholding entries each indicating the rating score for a particularcombination of a query term and an item identifier. The rating tablepreferably has no entries when it is initialized. In step 202, thefacility identifies all of the query result item selections made byusers during the period of time for which the rating table is beinggenerated. The rating table may be generated for the queries occurringduring a period of time such as a day, a week, or month. This group ofqueries is termed a “rating set” of queries. The facility alsoidentifies the terms of the queries that produced these query results instep 202. Performance of step 202 is discussed in greater detail belowin conjunction with FIG. 7. In steps 204-208, the facility loops througheach item selection from a query result that was made by a user duringthe time period. In step 204, the facility identifies the terms used inthe query that produced the query result in which the item selectiontook place. In steps 205-207, the facility loops through each term inthe query.

In step 206, the facility increases the rating score in the rating tablecorresponding to the current term and item. Where an entry does not yetexist in the rating table for the term and item, the facility adds a newentry to the rating table for the term and item. Increasing the ratingscore preferably involves adding an increment value, such as 1, to theexisting rating score for the term and item. In some embodiments, thefacility may add varying increment values in step 206 depending uponaspects of the current item selection. As one example, some embodimentsof the facility make a determination of the amount of effort required bythe user to make each selection, and base the increment value added instep 206 on that determination. For example, the selection of a firstitem that is three times further from the beginning of the query resultthan is a second item may result in an increment value for the selectionof the first item that is three times as large as an increment value forthe selection of the second item. Increment values for the selection ofitems that are reached by traversing additional links may likewiseexceed increment values for selections of items that can be displayedwithout selecting intermediate links. Aspects relating to thedetermination of the level of effort required for the user to select anitem in a query result are discussed further below in conjunction withFIGS. 10-13.

In step 207, if additional terms remain to be processed, the facilityloops back to step 205 to process the next term in the query, else thefacility continues in step 208. In step 208, if additional itemselections remain to be processed, then the facility loops back to step203 to process the next item selection, else these steps conclude.

FIGS. 3 and 4 are table diagrams showing augmentation of an item ratingtable in accordance with step 206 (FIG. 2). FIG. 3 shows the state ofthe item rating table before its augmentation. It can be seen that thetable 300 contains a number of entries, including entries 301-306. Eachentry contains the rating score for a particular combination of a queryterm and an item identifier. For example, entry 302 identifies the score“22” for the term “dynamics” the item identifier “1883823064”. It can beseen by examining entries 301-303 that, in query results produced fromqueries including the term “dynamics”, the item having item identifier“1883823064” has been selected by users more frequently than the itemhaving item identifier “9676530409”, and much more frequently than theitem having item identifier “0801062272”. In additional embodiments, thefacility uses various other data structures to store the rating scores,such as sparse arrays.

In augmenting the item rating table 300, the facility identifies theselection of the item having item identifier “1883823064” from a queryresult produced by a query specifying the query terms “human” and“dynamics”. FIG. 4 shows the state of the item rating table after theitem rating table is augmented by the facility to reflect thisselection. It can be seen by comparing entry 405 in item rating table400 to entry 305 in item rating table 300 that the facility hasincremented the score for this entry from “45” to “46”. Similarly, thefacility has incremented the rating score for this item identifier theterm “dynamics” from “22” to “23”. Although the increment valuesreflected in the differences between FIGS. 3 and 4 are both 1, as notedabove, different increment values may be used for different itemselections. The facility augments the rating table in a similar mannerfor the other selections from query results that it identifies duringthe time period.

Rather than generating a new rating table from scratch using the stepsshown in FIG. 2 each time new selection information becomes available,the facility preferably generates and maintains separate rating tablesfor different constituent time periods, of a relatively short length,such as one day. Each time a rating table is generated for a newconstituent time period, the facility preferably combines this newrating table with existing rating tables for earlier constituent timeperiods to form a rating table for a longer composite period of time.FIG. 5 is a table diagram showing the generation of rating tables forcomposite periods of time from rating tables for constituent periods oftime. It can be seen in FIG. 5 that rating tables 501-506 eachcorrespond to a single day between 8-Feb-98 and 13-Feb-98. Each time anew constituent period is completed, the facility generates a new ratingtable reflecting the user selections made during that constituentperiod. For example, at the end of 12-Feb-98, the facility generatesrating table 505, which reflects all of the user selections occurringduring 12-Feb-98. After the facility generates a new rating table for acompleted constituent period, the facility also generates a new ratingtable for a composite period ending with that constituent period. Forexample, after generating the rating table 505 for the constituentperiod 12-Feb-98, the facility generates rating table 515 for thecomposite period 8-Feb-98 to 12-Feb-98. The facility preferablygenerates such a rating table for a composite period by combining theentries of the rating tables for the constituent periods making up thecomposite period, and combining the scores of corresponding entries, forexample, by summing them. In one preferred embodiment, the scores andrating tables for more recent constituent periods are weighted moreheavily than those in rating tables for less recent constituent periods.When ranking query results, the rating table for the most recentcomposite period is preferably used. That is, until rating table 516 canbe generated, the facility preferably uses rating table 515 to rankquery results. After rating table 516 is generated, the facilitypreferably uses rating table 516 to rank query results. The lengths ofboth constituent periods and composite periods are preferablyconfigurable.

FIG. 6 is a table diagram showing a rating table for a composite period.By comparing the item rating table 600 shown in FIG. 6 to item ratingtable 400 shown in FIG. 4, it can be seen that the contents of ratingtable 600 constitute the combination of the contents of rating table 400with several other rating tables for constituent periods. For example,the score for entry 602 is “116”, or about five times the score forcorresponding entry 402. Further, although rating table 400 does notcontain an entry for the term “dynamics” and the item identifier“1887650024”, entry 607 has been added to table 600 for this combinationof term and item identifier, as a corresponding entry occurs in a ratingtable for one of the other constituent periods within the compositeperiod.

The process used by the facility to identify user selections isdependent upon both the kind of selection action used by the facilityand the manner in which the data relating to such selection actions isstored. One preferred embodiment uses as its selection action requeststo display more information about items identified in query results. Inthis embodiment, the facility extracts this information from logsgenerated by a web server that generates query results for a user usinga web client, and allows the user to select an item with the web clientin order display additional information about it. A web server generallymaintains a log detailing of all the HTTP requests that it has receivedfrom web clients and responded to. Such a log is generally made up ofentries, each containing information about a different HTTP request.Such logs are generally organized chronologically. Log Entry 1 below isa sample log entry showing an HTTP request submitted by a web client onbehalf of the user that submits a query.

-   1. Friday, 13-Feb-98 16:59:27-   2. User Identifier=82707238671-   3. HTTP REFERER=http://www.amazon.com/book_query_page-   4. PATH INFO=/book_query-   5. author=“Seagal”-   6. title=“Human Dynamics”

Log Entry 1

It can be seen by the occurrence of the keyword “book_query” in the“PATH_INFO” line 4 of Log Entry 1 that this log entry corresponds to auser's submission of a query. It further can be seen in term lines 5 and6 that the query includes the terms “Seagal”. “Human”, and “Dynamics”.In line 2, the entry further contains a user identifier corresponding tothe identity of the user and, in some embodiments, also to thisparticular interaction with the web server.

In response to receiving the HTTP request documented in Log Entry 1, thequery server generates a query result for the query and returns it tothe web client submitting the query. Later the user selects an itemidentified in the query result, and the web client submits another HTTPrequest to display detailed information about the selected item. LogEntry 2, which occurs at a point after Log Entry 1 in the log, describesthis second HTTP request.

-   1. Friday, 13-Feb-98 17:02:39-   2. User Identifier=82707238671-   3. HTTP REFERER=http://www.amazon.com/book_query-   4. PATH_INFO=/ISBN=1883823064

Log Entry 2

By comparing the user identifier in line 2 of Log Entry 2 to the useridentifier in line 2 of Log Entry 1, it can be seen that these logentries correspond to the same user and time frame. In the “PATH_INFO”line 4 of Log Entry 2, it can be seen that the user has selected an itemhaving item identifier (“ISBN”) “1883823064”. It can further be seenfrom the occurrence of the keyword “book_query” on the “HTTP_REFERER”line 3 that the selection of this item was from a query result.

Where information about user selections is stored in web server logssuch as those discussed above, the facility preferably identifies userselections by traversing these logs. Such traversal can occur either ina batch processing mode after a log for a specific period of time hasbeen completely generated, or in a real-time processing mode so that logentries are processed as soon as they are generated.

FIG. 7 is a flow diagram showing the steps preferably performed by thefacility in order to identify user selections within a web server log.In step 701, the facility positions a first pointer at the top, orbeginning, of the log. The facility then repeats steps 702-708 until thefirst pointer reaches the end of the log. In step 703, the facilitytraverses forward with the first pointer to the next item selectionevent. In terms of the log entry shown above, step 703 involvestraversing forward through log entries until one is found that containsin its “HTTP_REFERER” line a keyword denoting a search entry, such as“book_query”. In step 704, the facility extracts from this itemselection event the identity of the item that was selected and sessionidentifier that identifies the user that selected the item. In terms ofthe log entries above, this involves reading the ten-digit numberfollowing the string “ISBN=” in the “PATH_INFO” line of the log entryand reading the user identifier from the “User Identifier” line of thelog entry. Thus, in Log Entry 2, the facility extracts item identifier“1883823064” and session identifier “82707238761”. In step 705, thefacility synchronizes the position of the second pointer with theposition of the first pointer. That is, the facility makes the secondpointer point to the same log entry as the first pointer. In step 706,the facility traverses backwards with the second pointer to a queryevent having a matching user identifier. In terms of the log entriesabove, the facility traverses backward to the log entry having thekeyword “book_query” in its “PATH_INFO” line, and having a matching useridentifier on its “User Identifier” line. In step 707, the facilityextracts from the query event to which the second pointer points theterms of the query. In terms of the query log entries above, thefacility extracts the quoted words from the query log entry to which thesecond pointer points, in the lines after the “PATH_INFO” line. Thus, inLog Entry 1, the facility extracts the terms “Seagal”, “Human”, and“Dynamics”. In step 708, if the first pointer has not yet reached theend of the log, then the facility loops back to step 702 to continueprocessing the log, else these steps conclude.

When other selection actions are used by the facility, extractinginformation about the selection from the web server log can be somewhatmore involved. For example, where the facility uses purchase of the itemas the selection action, instead of identifying a log entry describing arequest by the user for more information about an item, like Log Entry1, the facility instead identifies a log entry describing a request topurchase items in a “shopping basket.” The facility then traversesbackwards in the log, using the entries describing requests to add itemsto and remove items from the shopping basket to determine which itemswere in the shopping basket at the time of the request to purchase. Thefacility then continues traversing backward in the log to identify thelog entry describing the query, like Log Entry 2, and to extract thesearch terms.

Rather than relying solely on a web server log where item purchase isthe selection action that is used by the facility, the facilityalternatively uses a database separate from the web server log todetermine which items are purchased in each purchase transaction. Thisinformation from the database is then matched up with the log entrycontaining the query terms for the query from which item is selected forpurchase. This hybrid approach, using the web server logs and a separatedatabase, may be used for any of the different kinds of selectionactions. Additionally, where a database separate from the web server logcontains all the information necessary to augment the rating table, thefacility may use the database exclusively, and avoid traversing the webserver log.

The facility uses rating tables that it has generated to generateranking values for items in new query results. FIG. 8 is a flow diagramshowing the steps preferably performed by the facility to order a queryresult using a rating table by generating a ranking value for each itemin the query result. In steps 801-807, the facility loops through eachitem identified in the query result. In step 802, the facilityinitializes a ranking value for the current item. In steps 803-805, thefacility loops through each term occurring in the query. In step 804,the facility determines the rating score contained by the mostrecently-generated rating table for the current term and item. In step805, if any terms of the query remain to be processed, then the facilityloops up to step 803, else the facility continues in step 806. In step806, the facility combines the scores for the current item to generate aranking value for the item. As an example, with reference to FIG. 6, inprocessing datum having item identifier “1883823064”, the facilitycombines the score “116” extracted from entry 602 for this item and theterm “dynamics”, and the score “211” extracted from entry 605 for thisitem and the term “human”. Step 806 preferably involves summing thesescores. These scores may be combined in other ways, however. Inparticular, scores may be adjusted to more directly reflect the numberof query terms that are matched by the item, so that items that matchmore query terms than others are favored in the ranking. In step 807, ifany items remain to be processed, the facility loops back to step 801 toprocess the next item, else the facility continues in step 808. In step808, the facility displays the items identified in the query result inaccordance with the ranking values generated for the items in step 806.Step 808 preferably involves sorting the items in the query result indecreasing order of their ranking values, and/or subsetting the items inthe query result to include only those items above a threshold rankingvalue, or only a predetermined number of items having the highestranking values. After step 808, these steps conclude.

FIG. 9 is a flow diagram showing the steps preferably performed by thefacility to select a few items in a query result having the highestranking values using a rating table. In steps 901-903, the facilityloops through each term in the query. In step 902, the facilityidentities among the table entries for the current term and thoseentries having the three highest rating scores. For example, withreference to FIG. 6, if the only entries in item rating table 600 forthe term “dynamics” are entries 601, 602, 603, and 607, the facilitywould identify entries 601, 602, and 603, which are the entries for theterm “dynamics” having the three highest rating scores. In additionalpreferred embodiments, a small number of table entries other than threeis used. In step 903, if additional terms remain in the query to beprocessed, then the facility loops back to step 901 to process the nextterm in the query, else the facility continues in step 904. In steps904-906, the facility loops through each unique item among theidentified entries. In step 905, the facility combines all of the scoresfor the item among the identified entries. In step 906, if additionalunique items remain among the identified entries to be processed, thenthe facility loops back to step 904 to process the next unique item,else the facility continues in step 907. As an example, if, in itemrating table 600, the facility selected entries 601, 602, and 603 forthe term “dynamics”, and selected entries 604, 605, and 606 for the term“human”, then the facility would combine the scores “116” and “211” forthe item having item identifier “1883823064”, and would use thefollowing single scores for the remaining item identifiers: “77” for theitem having item identifier “0814403484”, “45” for the item having itemidentifier “9676530409”, “12” for the item having item identifier“6303702473”, and “4” for the item having item identifier “0801062272”.In step 907, the facility selects for prominent display items having thetop three combined scores. In additional embodiments, the facilityselects a small number of items having the top combined scores that isother than three. In the example discussed above, the facility wouldselect for prominent display the items having item identifiers“1883823064”, “0814403484”, and “9676530409”. Because the facility instep 907 selects items without regard for their presence in the queryresult, the facility may select items that are not in the query result.This aspect of this embodiment is particularly advantageous insituations in which a complete query result is not available when thefacility is invoked. Such as the case, for instance, where the queryserver only provides a portion of the items satisfying the query at atime. This aspect of the invention is further advantageous in that, byselecting items without regard for their presence in the query result,the facility is able to select and display to the user items relating tothe query even where the query result is empty, i.e., when no itemscompletely satisfy the query. After step 907, these steps conclude.

FIGS. 10-13 are display diagrams showing examples of considerations usedby embodiments of the facility to determine the level of effort expendedby the user to select an item from a query, result. FIG. 10 is a displaydiagram showing an initial query result display. This display 1010 inbrowser window 1000 shows the top portion of the first page of amultiple-pale query result. The position of scrollbar elevator 1001 atthe top of its scroll bar indicates that this web page is scrolled to aposition at the top of the web page. This scrolling position istypically the one at which web pages initially display. The displaycontains the first three items in the query result, items 1011, 1012,and 1013. In general, the first item 1011 is regarded as the easiestitem for the user to select, as it is spatially the nearest item to thebeginning of the first page of the query result. Accordingly, in manyembodiments, a relatively small increment value is added to ratingscores for the selection of this item in this query result. The otheritems on this display may either be regarded as requiring the sameamount of effort to select, since selecting them does not requirescrolling the display, or as requiring a slightly higher level of effortto select, because the user must read through one or more other items inthe query result to reach these items.

FIG. 11 is a display diagram showing a second display of the samplequery result shown in FIG. 10. This display 1110 shown in browser window1100 is generated by scrolling down one screen from display 1010 shownin FIG. 10, such as by pressing a PageDown key on the keyboard or byclicking the portion of the scrollbar beneath scrollbar elevator 1001.This display 1110 contains the next three items in the query result,items 1111, 1112, and 1113. These three items are typically regarded asrequiring more effort for the user to select, as such selection involvesscrolling and additional reading not required to select items 1011-1013shown in FIG. 10.

FIG. 12 is a display diagram showing a third display of the query resultshown in FIG. 10. From the relatively low position of scrollbar elevator1201, it can be seen that this display 1210 shown in browser window 1200is a screen near the bottom of first page of the query result, which isdisplayed by further scrolling the page down. Because this larger amountof reading and/or scrolling is required to select one of the displayeditems 1211-1213, selecting one of these items typically producessignificantly larger increment value than selecting items 1011-1013 or1111-1113. The display further indicates that this first page of thequery result contains the first 25 items of this query result (1221) outof a total of 54 items contained by the entire query result (1222). Toreach the next page of this query result, the user can click button 1231to display a second page of the query result beginning with the 26thitem of the query result.

FIG. 13 is a display diagram showing a fourth display of the samplequery result. Display 1310 is the top of the second page of the samplequery result, displayed by clicking button 1231 shown in FIG. 12. Thedisplay contains items 1311-1313. Selection of any of these items istypically associated with an interval value greater than that for any ofthe above-discussed items, as reaching this display requires asignificant amount of scrolling as well as selecting an intermediatebutton, also called a “link.”

Different embodiments of the facility take various approaches todetermining the amount of effort required to select an item in a queryresult. In some embodiments, the facility determines the amount ofeffort required based upon the number of items that precede the selecteditem in the query result. In other embodiments, the facility makes thisdetermination based upon how far down on a query result page theselected item occurs (such as in distance, words, or characters), and/orbased upon whether the selected item occurs on a page after the firstpage of the query result. In other embodiments, the facility uses otherapproaches to determining the amount of effort needed to select an itemfrom a query result that correspond to various other techniques fornavigating a query result.

The facility also uses various mechanisms for performing thisdetermination, including determining the number of items that precedethe selected item in the query result, and/or monitoring userinteractions that navigate to the selected item within the query result.Such monitoring may encompass monitoring user interface interactions,such as keystrokes, mouse clicks, scroll wheel rotations, and voicecommands, as well as by monitoring web server requests corresponding tonavigational functions.

In some embodiments, the facility uses stemming techniques to combinerating scores for query terms having the same root as query termsoccurring in the query. In different embodiments of the facility, thestemming techniques are incorporated in different ways. As a firstexample, in the item rating table shown in FIGS. 3 and 4, the termcolumn containing terms occurring in queries may be replaced with a termroot column containing the roots of the terns occurring in queries. Forinstance, when items are selected from a query containing the terms“human” and “dynamics,” the facility would increment scores for itemrating table rows containing the term roots “human” and “dynamic”—theroots obtained by stemming those terms.

In a second example, the facility expands the terms occurring in a queryfrom whose query result an item is selected to all of the differentterms that share the same root as the term occurring in the query. Forexample, for a query containing the term “dynamics,” the facility wouldincrement the score for rows with the item rating table containing theterms “dynamic,” “dynamics,” “dynamism,” “dynamist,” “dynamistic,” and“dynamically.”

In a third example, the facility continues to update the item ratingtable without any use of stemming as described above, but in reading theitem rating table, such as in step 804, the facility combines, for eachterm occurring in the query at issue, the scores for all of the termsbearing the same root with the terms occurring in the query. Forexample, if the facility received a query containing the term“dynamics,” the facility would combine with the score for this term thescores for the additional terms “dynamic,” “dynamism,” “dynamist,”“dynamistic,” and “dynamically,” which all share the root “dynam-.”Other approaches to utilizing stemming are part of additionalembodiments of the facility.

While the present invention has been shown and described with referenceto preferred embodiments, it will be understood by those skilled in theart that various changes or modifications in form and detail may be madewithout departing from the scope of the invention. For example, thefacility may be used to rank query results of all types. The facilitymay use various formulae to determine in the case of each itemselection, the amount by which to augment rating scores with respect tothe selection. Further, the facility may employ various formulae tocombine rating scores into a ranking value for an item. The facility mayalso use a variety of different kinds of selection actions to augmentthe rating table, and may augment the rating table for more than onekind of selection action at a time. Additionally, the facility, mayaugment the rating table to reflect selections by users other than humanusers, such as software agents or other types of artificial users.

1-20. (canceled)
 21. A method in a computing system for ranking items in a query result, the method comprising: receiving a query specifying one or more terms; generating a query result identifying items satisfying the query; for each of the items identified in the query result, combining ratings reflecting both (a) the frequencies with which users selected the item in query results produced for earlier queries specifying one or more terms of the query and (b) levels of effort required to make such selections, such that the combination of ratings produces a ranking value for the item; and displaying the items identified by the generated query result in accordance with their produced ranking values.
 22. The method of claim 21 wherein the ratings that are combined reflect the number of items that preceded the selected items in the query results from which they were selected.
 23. The method of claim 21 wherein the ratings that are combined reflect the set of navigation commands needed to reach the selected items in the query results from which they were selected.
 24. The method of claim 21 wherein the ratings that are combined reflect the extent to which the query results from which the items were selected were scrolled to reach the selected items.
 25. The method of claim 21 wherein the ratings that are combined reflect, in the query results from which the items were selected, the number of pages of each query result that preceded the page of each query result containing the selected item.
 26. The method of claim 21 wherein the ratings that are combined reflect the position of the item in query results produced for earlier queries.
 27. The method of claim 21 wherein displaying the items from the generated query result includes sorting the items by their ranking values.
 28. The method of claim 21 wherein displaying the items from the generated query result includes displaying items with a ranking value greater than a predefined threshold.
 29. A computer-readable medium containing instructions for compiling statistics usable to rank items in a distinguished query result produced for a distinguished query, by a method comprising: receiving a rating set of queries, each query in the rating set specifying one or more terms; for each query in the rating set, generating a query result identifying one or more items satisfying the query; allowing a user to select one or more of the items identified in the query result; and for items selected from the query result, for terms specified by the query, determining an adjustment factor indicating the level of effort necessary to effectuate selection of the item; adjusting a rating score corresponding to the combination of the selected item and the term specified by the query by the determined adjustment factor; and storing the rating score.
 30. The computer-readable medium of claim 29 wherein determining each adjustment factor includes increasing the adjustment factor for items occurring in the query result before the selected item.
 31. The computer-readable medium of claim 29 wherein determining each adjustment factor includes increasing the adjustment factor for navigation operations performed to reach the selected item in the query result.
 32. The computer-readable medium of claim 29 wherein determining each adjustment factor reflects the position of the item in query results produced for earlier queries.
 33. The computer-readable medium of claim 29 wherein determining each adjustment factor reflects the number of items that preceded the selected items in the query results from which they were selected.
 34. The computer-readable medium of claim 29 wherein determining each adjustment factor reflects the set of navigation commands needed to reach the selected items in the query results from which they were selected.
 35. The computer-readable medium of claim 29 wherein determining each adjustment factor reflects the extent to which the query results from which the items were selected were scrolled to reach the selected items.
 36. The computer-readable medium of claim 29 wherein determining each adjustment factor reflects, in the query results from which the items were selected, the number of pages of each query result that preceded the page of each query result containing the selected item.
 37. The computer-readable medium of claim 29 further comprising displaying the items in the distinguished query result in accordance with their rating scores.
 38. One or more computer memories collectively containing a user behavior data structure usable to rank the relevance of items in a query result, the data structure comprising: a plurality of rating scores, wherein each rating score corresponds both to a query term and to an item, and reflects quantitatively the extent to which users have selected the item from query results generated for earlier queries specifying the query term and the level of effort required to make such selections, such that the data structure may be used to rank items in a distinguished query result produced for a distinguished query by, for each item in the distinguished query result, retrieving from the data structure the rating scores corresponding to the item and any term specified in the distinguished query and combining the retrieved rating scores to generate a ranking value for the item.
 39. The data structure of claim 38 wherein the level of effort reflects the position of the item in query results produced for earlier queries.
 40. The data structure of claim 38 wherein the level of effort reflects the set of navigation commands needed to reach the selected items in the query results from which they were selected.
 41. A method in a computing system for ranking items in a query result, the method comprising: receiving a query specifying one or more terms; generating a query result identifying items satisfying the query; for each of the items identified in the query result, generating a ranking value reflecting both (a) the number of users that selected the item in query results produced for earlier queries specifying one or more terms of the query and (b) a measure of the difficulty of making such selections; and displaying the items identified by the generated query result in accordance with their generated ranking values.
 42. A method in a computing system for ranking items in a query result, the method comprising: receiving a query specifying one or more terms; generating a query result identifying a plurality of items satisfying the query; for a plurality of items identified in the query result, combining ratings of frequencies with which users selected the item in earlier queries specifying either (a) one or more terms of the query or (b) a term sharing a root with a term of the query to produce a ranking value for the item; and displaying the items identified by the generated query result in accordance with their produced ranking values.
 43. A computer-readable medium whose contents cause a computing system to rank items in a query result by: receiving a query specifying one or more terms; generating a query result identifying a plurality of items satisfying the query; for a plurality of items identified in the query result, combining ratings of frequencies with which users selected the item in earlier queries specifying either (a) one or more terms of the query or (b) a term sharing a root with a term of the query to produce a ranking value for the item; and displaying the items identified by the generated query result in accordance with their produced ranking values. 